Gilbert was the son of Richard the Fearless. As a young man he inherited Brionne and became one of the most powerful landowners in Normandy.

In about 1026, Herleva of Falaise, the sixteen year old daughter of a tanner from Falaise in Normandy, gave birth to a son called Richard. Gilbert was the father of the child. As Herleva was not married to Gilbert, the boy became known as Richard Fitz Gilbert. The term 'Fitz' was used to show that Richard was the illegitimate son of Gilbert.

When Robert, Duke of Normandy died in 1035 William the Conqueror inherited his father's title. Several leading Normans, including Gilbert of Brionne, Osbern the Seneschal and Alan of Brittany, became William's guardians.

A number of Norman barons would not accept an illegitimate son as their leader and in 1040 an attempt was made to kill William. The plot failed but they did manage to kill Gilbert of Brionne. It is believed his killer was Ralph of Wacy.