If the entrance is fitted with doorpost and lintel, it is no different than any other room. If it is closed at its back, the mezuzah is placed on the right side of one entering from the big room.
If there is a door at its rear, one would decide based on the priorities of right-hand placement discussed in General Principles. If it is determined that this rear alcove door is its entrance, the mezuzah is placed on the right side of one walking through it and into the big room. Since one goes from this alcove into the big room, the two areas are considered to be one room, and the mezuzah on the alcove entrance suffices for the opening into the big room as well,[A]
unless it is fitted with its own posts and lintel.[1] This is a common design of an entrance hall.
If an alcove is formed by an indentation in the walls, and it has no doorposts besides the corners where its walls continue into the larger room, the area requires no mezuzah as it is only an extension of the big room.[2][B]
[1] Agur B’ohalecha 21:26; Cf. Chovas HaDar 7:7, who rules that a mezuzah should be placed on the end of alcove on the right as it is an entrance to the big room.
[2] Agur B’ohalecha 21:25 exempts this opening entirely as he deems the area to be merely an extension of the big room. Chovas HaDar 7:7 also exempts it unless it has an entrance from its back. Cf. Pitchei She’arim 287:12,17, who recommends affixing a mezuzah without a berachah on the right of one entering the alcove, if the left side is also a corner and not just a continuous wall.