What you want to do requires two steps to follow:
1. The common way to add javascript code to an event handler (like onclick) is to use the Attributes collection of your WebControl. For instance, if you have a button  all you have to do to open a new window  when you click  on it is write this in the Page_Load:

[c#]
myButton.Attributes["onclick"] = "window.open([url], [hwnd], [features]);";

Remember that this button should not fire postback so you won't be able to use a Button (<asp:Button>), just use the <input type="button"> and write a runat server attribute. This way you will be using a HtmlInputButton class not a Button class and will be able to not generate a postback.

Also, you can take a look at this server control  from Andy Smith called RemoteWindow [1] which will do this job for you but will render an HyperLink, not a button

2. In the popup you will have to use window.opener to set anything you want to the parent Asp.Net page. What I would do here is reload the parent page with a querystring value generated by the popup that will indicate the dynamic creation of the checkbox  when the page loads.

Surely, you can do it without refreshing, but it will require more hacks and cross-browser issues to create a checkbox that would be able to be read on a future postback of the page.