The earliest stars and proto-galaxies in the Universe are thought to have formed
within the first few hundred million years after the Big Bang.  However, these
sources are faint and at such distances, immensely challenging to detect
directly.  Gamma-ray bursts, thanks to their immense luminosities, can easily
be seen at very high redshifts, and so offer an alternative route to exploring the
early formation of structure in the Universe.  They act as beacons of massive star formation
and offer the chance to probe the properties and environments of these early galaxies,
and their effects on the intergalactic medium.  The recently discovered GRB 090423, which at
redshift 8.2 is the most distant spectroscopically confirmed object, opens the
door to such studies. I will review what we have learnt from that event and other high
redshift GRBs, and consider prospects for the future.