You can restrict the allowed companies for each user under "Settings" > "Users" > "User Name". Under the tab "Access Rights" you can select which companies are allowed for this user. If a user is allowed multiple companies you can (de)activate companies left of the username in the top right corner. Also make sure that the contacts are assigned to the correct company. I hope this answers your question.
Unluckily my problem of contact visibility is still there, but luckily I eventually probably found out the confusing aspect of this issue.
I would believe the Contact visibility of Odoo in a multi-companies setup is working.
The reason users still able to see contacts not belonged to their company because the 'Company' field I seen on my Odoo setup IS NOT the 'Company' required to distinguished between companies in Odoo.
Why I know?
When I create a contact via Contact Form, this contact is visible/invisible properly among companies.
However, when I import contacts from xlsx with the field 'Company' as I seen, these contacts visible to all users.
That means the 'Company' seen is not the same as the Odoo 'Company' required, (or an additional field required).
So what help?
I create a contact from Odoo Contact Form and export that contact from Action->Export and select required fields to export, there are many Company field for selection, will you know which 'Company' is the 'Company' required by Odoo security control?