The gallery was initially housed in the District Museum and in 1965 adapted the house on The Trinity Street which was assigned to it. In late 1965, when the construction work was finished, it was decided to dispose of the East Bohemian Publishing House in Havlickuv Brod and its building – Malina’s house, where also the exhibition hall of the publishing house was – was offered to the gallery. That building was situated more preferably and it was one of the most beautiful town houses in the town. And thus, the gallery once again decided to go through the purgatory of another adaptation and acquired the late Gothic house from the 2nd half of the 15th century on the square, which was in its history adapted in the Renaissance and Baroque style. The house was opened to the public in December 1966, the gallery was housed there until August 1999 when it moved to the newly refurbished town house No. 18 on Havlicek Square.
Gallery's new building is a house in the past generally referred to as “the old pharmacy.” It neighbours the memorable Havlicek’s House and is on the boundary of the square and Horní Street, but already in the oldest city books it is known as a "house in the market square", where the former ancestral homes of town’s leading patrician families were. Therefore it is very surprising and unusual in other houses that it frequently changed its owners, especially in the 18th century.
The oldest known record relevant to this house is from 1581, when the scribe Jan Bydžovský bought this house from Václav Chmelík.
In 1626 in Horní Street near the gate a great fire broke out and, among many other buildings, it burned down this house as well. The former owner Jiří Krupský newly built the house again. Another in a series of frequently changing owners was Lukáš Kobzin, a younger brother in law of Kateřina Barbara Kobzinová. In 1755 the house was purchased by a pharmacist Jan Antonín Wetzl who expensively rebuilt the house and turned it into a richly and for that time very efficiently furnished pharmacy. Also a modification of the house façade comes from this period. All the subsequent owners were pharmacists, and that is the reason why the house received this designation.
In the second half of the 20th century, the house was managed by a company Textil Pardubice which had a shop and warehoses on the first floor. On the second floor were two residential units. Gallery bought the house in March 1997 and began a costly reconstruction. The reconstruction took place in the years 1997–1999 and compared to the original project it required a higher cost because after removing plasters uncovered the poor static conditions of the building. It was, for example, necessary to statically strengthen all the vaults in the premises of the basement, first and second floor, re-brick the supporting wall on which there were vaulted two rooms and which was based on a rotten oak beam bulk embankment on the first floor vaults. After knocking off the plasters, there were found numerous quality Gothic and Renaissance architectural elements that remain exposed after restoration and serve as a proof of the very high status that the house and its owner occupied in the Middle Ages. This is also underscored by the archaeological findings, most important of which are fragments of Gothic castellated tiles or fragments of green glazed chamber tiles with figural and architectural motifs. All these discoveries are deposited in the Museum of Vysocina in Havlickuv Brod and are occasionally represented to the visitors of the gallery.
The gallery moved to the new building in August 1999 and in October it publicized its exhibition spaces. The purchase and complete renovation of the building granted the gallery an adequate replacement for the existing building. It enabled the convenient separation of the permanent exposition halls from the exhibition halls and the use of the renovated attic also a part of the missing quality background. It also made possible the barrier-free access to the exhibition space on the first and second floors and to the permanent exhibition hall on the second floor.