The gas surrounding first-generation (Population III, hereafter Pop III) stars is expected to emit a distinct signature in the form of the He II recombination line at 1640 Å(He II λ1640). Here we explore the challenges and opportunities in identifying this elusive stellar population via the He II λ1640 in M <SUB>⋆</SUB> > 10<SUP>7.5</SUP> M <SUB>⊙</SUB> galaxies during the Epoch of Reionization (z ≃ 6–10), using JWST/NIRSpec. With this aim in mind, we combine cosmological dustyGadget simulations with analytical modeling of the intrinsic He II emission. While tentative candidates with bright He II emission like GN-z11 have been proposed in the literature, the prevalence of such bright systems remains unclear due to significant uncertainties involved in the prediction of the He II luminosity. In fact, similar Pop III clumps might be almost 2 orders of magnitude fainter, primarily depending on the assumed Pop III formation efficiency and initial mass function in star-forming clouds, while the effect of stellar mass loss is responsible for a factor of order unity. Moreover, up to ∼90% of these clumps might be missed with NIRSpec multi-object spectroscopy due to the limited field of view, while this problem appears to be less severe with NIRSpec's integral field unit. We investigate the potential of deep spectroscopy targeting peripheral Pop III clumps around bright, massive galaxies to achieve a clear detection of the first stars.