Brixton is definitely not a place that you'd wanna be making enemies, especially not in a racial way.
I went to a street party there a few years ago and shit was one of the best days of my life. Guys serving rum and juice in cups, nangs galore, mad reggae music, dope food. Things got pretty heated at a couple of points however and the riot squad had to be called in. I think it's been cancelled now because of the security risk and level of mess left behind.