When you click on the name of the user, does the manage access page opens up?
If yes, click on the permission below. For me it says, "Direct access: can view". In the drop down, there is a remove option.
Refer the screenshot. Let me know if this does not work.