Cut the pound cake into slices about ⅓” thick crosswise.
Cover the bottom of a 2-3 quart dish (straight-sided glass dish is considered the “proper” dish for a trifle, but really anything will do) with slices of pound cake. Break the slices as necessary to make an even layer, filling in all holes with cake bits.
Sprinkle 2 tablespoons brandy over the cake.
Spread 1 cup of the custard over that evenly, reaching edges. Dollop a little of the jam over that and half the berries.
Make another layer of cake, brandy, custard, jam, berries.
If you have enough cake to make a final layer, do so. Spread the remaining custard, jam, berries over that.
Cover the top of the trifle with whipped cream and sprinkle with almonds.
Serve right away if you must, but it is better if allowed to sit in the refrigerator for an hour or two before serving.