Your randomart contains only information generated from your public key. For you to receive data from someone they need your public key. If they have your public key then they will also be able to generate your randomart for themselves.
TL;DR: Your key, and therefore randomart, is "public" information and should be fine to broadcast. Your broadcast copy of the randomart can be used for someone to compare what they have generated to quickly check they have received the right key. This is obviously subject to the key being stored securely or verified by some other means as them simply downloading the key from the same location as the randomart does not stop both parts having been tampered with in that location.
If you can see the randomart then you already have access to the entire public key. The randomart does nothing more than walk through the bit data of the public key and draw what is essentially a kind of heatmap of where a "worm" might travel using a set of rules based on that data.
It is nothing more than a visual representation of the data, it does not expose anything about the secret key, and is nothing more than a way to visually verify a familiar public key.
The fact is that you need the entire public key to communicate with the host in the first place. You have to use the public key to encrypt data to send to the host, which then uses its secret key to decrypt it. In return you may provide someone with your public key for them to provide you things that can be decrypted with your secret key.
The randomart is based only on the actual data bytes within the public key and is generated "on the fly".
From Security SE: How secure is Visual host key, and how is SHA2 converted into that representation?
Start in the middle, and use the hash as a kind of treasure map for where to move next. For each step, take the next two bits in the hash. 00 means move northwest; 01 means northeast; 10 means southwest; 11 means southeast. Hug the wall if you bump into it. Count the number of times you step on each square.
Finally, for each square you stepped on, choose a symbol based on the number of times you stepped on the square, from the following list of 14 symbols: .o+=*BOX@%&#/^. (That's 15 symbols total if you include "" for unvisited squares.) Mark your start and end points specially, with an S and an E, respectively.
It is not guaranteed to be unique, but can be used to verify a key is what is expected to be seen.
It is no more or less secure than the public key itself. If your key is weak or compromised somehow (someone else has your secret key) then the randomart won't change a thing.