Lay the sponge pieces and biscuits in the base of a deep glass dish. Spoon over the sherry and cognac and leave to soak in for a few minutes.
Warm the jam until it is runny then pour over the soaked sponge and biscuits. Sprinkle the fruit and flaked almonds on top then pour over the custard.
To make the syllabub topping, place the sherry, brandy and lemon juice into a bowl and stir in the sugar until it dissolves.
Pour in the cream with a pinch of nutmeg. Whisk together using a wire balloon whisk until it holds its shape. Don't over-whisk, or whisk too vigorously, as the cream may split.
Spoon the syllabub over the trifle and chill, preferably overnight.
Just before serving, scatter with the grated orange rind and/or flaked toasted almonds.