I have encountered an error when trying to create an index, saying “Error forbidden”
turns out the issue is with elastic search.
checking elastic search, there is an error with operation blocked due to “read only”:
this not only blocking the kibana for updating, for example create new index, it blocks new logs from being pumped in as well.
and ultimately, this was set due to the flood_stage watermark:
to sort out the issue, i have cleared some old docker images and containers:
followed by a call to reset the read_only
which brought the elasticsearch back to normal
