Count Albert Alexander de Pourtalès was the son of the Neuchâtel-Prussian Count Jules Henri Charles Frédéric de Pourtalès (1779–1861) and Marie Louise Elisabeth (1793–1881), née Marquise Castellane-Norante, who had close ties to the Prussian royal and French imperial courts. Like his father, Albert worked in the service of Prussia and travelled across Europe and the East as a diplomat, which he knew very well thanks to three stays in Istanbul between 1848 and 1851, first as secretary of the Prussian legation and then, from 1850, as its envoy.
In 1844, Albert de Pourtalès acquired Oberhofen Castle and from 1848 remodelled it in collaboration with his father. In 1854, he entrusted the conception and execution of the Selamlik there, an Arab-style interior inspired by Ottoman houses in Cairo and Damascus, to the Bernese architect and draughtsman Theodor Zeerleder (1820–1868).